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ABSTRACT 

The objectives of this exploratory study were to 
determine: (1) the approach to learning of physics students (N=142) 
at John Abbott College (Quebec, Canada) as determined by the Study 
Process Questionnaire; (2) the intellectual demands of quizzes, 
tests, and final exams in physics using a scheme derived from Bloom's 
taxonomy; and (3) the relationships between the approach to learning, 
the intellectual demands of assessment, and the performance of the 
students. The findings show that most incoming physics students 
approach physics with the intention of memorizing formulae rather 
than understanding concepts. They adopt surface or surface-achieving 
approaches. In examining the intellectual demands of the quizzes, 
tests and final examinations it was found that the majority (70%) of 
items required routine problem solving, while 287. required 
comprehension. The grade assigned to items requiring comprehension 
increased from Mechanics 101 (197.) to Electricity and Magnetism 201 
(287.) to Waves and Optics 301 (327.). This approach to learning 
adopted by students was found to be related to the intellectual 
demands of the examinations, to the students* performances on the 
final examination, and to their prior knowledge of the concept of 
force. (Author/JRH) 
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Abstract 



The objectives of this exploratory study were to determine: 1) The 
approach to learning of physics students at John Abbott College as determined 
by the Study Process Questionnaire, 2) the intellectual demands of quizzes, 
tests and final exams in physics using a scheme derived from Bloom's 
taxonomy, and 3) the relationships between the approach to learning, the 
intellectual demands of assessment, and the performance of the students. 

The students in the study wrote the Study Process Questionnaire and 
the Force Concept Inventory in Mechanics 101 classes at the start of the 1993 
fall semester. Students who proceeded to Electricity and Magnetism 201 re- 
wrote the measures at the end of their second semester. 

The findings show that most incoming Physics students approach 
Physics with the intention of memorizing formulae rather than 
understanding concepts. They adopt surface or surface-achieving approaches. 
After two semesters the achieving approach had decreased while the surface 
motivation had increased and there was a trend for the deep motivation to 
decrease. 

In examining the intellectual demands of the quizzes, tests and final 
examinations it was found that the majority (70%) of items required routine 
problem solving, while 28% required comprehension. The grade assigned to 
items requiring comprehension increased from Mechanics 101 (19%)) to 
Electricity and Magnetism 201 (28%) to Waves and Optics 301 (32%-). 

The approach to learning adopted by students was found to be related 
to the intellectual demands of the examinations, to the students' 
performances on the final examinations, and to their prior knowledge of the 
concept of force. 
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Resume 

Cette etude exploratoire avait pour objet de determiner: 1) comment les 
etudiants au College John Abbott abordent l'apprentissage de la physique c\ 
l'aide du "Study Process Questionnaire", 2) les exigences intellectuelles que 
posent les examens modulaires et les examens de synthase bases sur la 
taxonomie de Bloom, et 3) les relations qui existent entre l'approche 
cognitive, les exigences intellectuelles de revaluation et la performance des 
etudiants. 

Les etudiants ayant participe au projet ont repondu au "Study Process 
Questionnaire" et au "Force Concept Inventory" qui leur ont ete soumis au 
debut du semestre d'automne 1993 durant le cours de Mecanique 101. Les 
etudiants ayant reussi ce cours et s'etant inscrits au cours d'Electricite et 
Magnetisme 201 ont repondu aux memes questions h la fin du second 
semestre. 

Nos resultats montrent que la plupart des etudiants qui commencent en 
physique abordent l'etude de cette discipline avec l'intention de memoriser 
les formules plutot que d'essayer de saisir les concepts. lis pref£rent une 
approche superficielle ou une approche superficielle de realisation. Nous 
avons constate qu'apr^s deux semestres, l'approche de realisation a diminue 
tandis que la motivation superficielle a augmente et qu'enfin une tendance 
decroissante se manifeste pour la motivation profonde. 

En examinant les exigences intellectuelles demandees aux examens 
modulaires et aux examens de synthase, nous avons decouvert que 70 % des 
questions n'exigeaient que de 1'habilete h resoudre des probl&mes de faqon 
routini^re tandis que 28 % exigeaient de la comprehension. La cote attribute 
aux questions exigeant de la comprehension a augmente de 19% durant le 
cours de Mecanique 101, h 28% durant le cours d'Electricite et Magnetisme 201 
et enfin h 32% durant le cours d'Optique et Ondes 301. 

Nous avons trouve que la faqon d'aborder l'etude de la physique etait 
reliee aux exigences intellectuelles demandees aux examens, aux resultats 
obtenus aux examens de synthase ainsi quaux connaissances anterieures sur 
le concept de force. 
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CHAPTER 1 



Approach To Learning And Assessment In Physics 

1.0 Introduction 

One of the most important and controversial issues in contemporary 
education is that of assessment: the assessment of student learning and the 
impact of assessment on student learning. When students enter their 
classrooms, they look to the teacher for guidance about what to learn and how 
to learn and, rightly or wrongly, they see the tests and other assessments as 
indicators of what the teachers consider to be important. The influence of 
final examinations is great. Indeed the primary concern voiced by most 
students facing a learning task is. "Is this going to be on the test?" After 
reviewing over 200 studies of the impact of classroom evaluation, Crooks 
(1988) concluded that assessment guided the student' r judgment of what it 
was important to learn, and affected their motivation and approach to 
studying; that is the hozv of their approach to the learning task. If the test 
focuses on factual knowledge, the student will learn to memorize; if the test 
requires analytical thinking the student will learn to reason analytically. 
Over twenty five years ago Rogers (1969), in speaking about learning physics, 
pointed out that learning will be sabotaged if the final exam asks for numbers 
to be put in memorized formulae even if the classes were dynamic, 



complemented by intriguing experiments and were accompanied by forceful 
exhortations to understand the physics. The intellectual skills the students 
rehearse will depend on the cognitive demands of the tasks they are asked to 
undertake. 

Once the teachers' expectations have been communicated the students 
can decide if they want to study and what learning strategies they want to use. 
The combination of strategy and motivation is called the approach to learning 
of the student. Three approaches have been identified; surface, deep, and 
achieving (Ramsden, 1991). In a surface approach the student focuses on 
memorizing to obtain a passing grade, a deep approach involves an intention 
to understand the material, while in an achieving approach a student adopts 
deep or surface level strategies according to what he or she judges to be most 
efficient for obtaining grades (Biggs, 1987). While students can control the 
approach to learning, they are just a part of a larger system. The boundaries of 
the system are set in part by the institution, in part by the participants 7 
perceptions of one another, and in part by the habits and practices of both 
teacher and student (Bhushan, 1991; Brekelmans, Wubbels, and Creton, 1990; 
Roth 1994). Within these boundaries are many complex interactions that 
influence the quality of learning: one of these is the interaction between the 
student's motives and strategies and the assessment practices of the teacher. 

1.1 Statement Of The Problem 

The purpose of the present study was threefold: first to determine the 
approach to learning of students in physics classes, as measured by the Study 
Process Questionnaire (Biggs, 1987); second to determine the intellectual 
demands of final examinations, tests, and quizzes in physics, using a scheme 
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based on Bloom's Taxonomy (Bloom, 1956); and third to search for 
relationships between these two variables, as well as the impact on the 
performance and persistence of the student. The performance of the students 
£ was measured by their grades in the physics final exams, and by their 

understanding of the concept of force as measured by their score on the Force 
Concept Inventory (Hestenes, Wells, and Swackhamer, 1992). 
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Three questions were asked:- 



What is the approach to learning, as assessed by the Study 
Process Questionnaire, of students studying physics at John 
Abbott College? 



£ 2 What are the intellectual demands of final examinations, 

tests, and quizzes, in physics at John Abbott College? 

i 

g 3 Are there relationships between the intellectual demands of 

the exam, performance on the exam, and the learning 
£ approach of the student? 



1.2 Review Of The Literature 



1.20 Persistence And Attrition 



In Quebec, 477, of those between 17 and 19 years old attend Cegep and 
of these only 19%. choose to study science. At John Abbott College more than 
the provincial average of entering students choose science (27% vs. l c n ) 



ERIC 



(John Abbott College Registrar's statistics for 1991; Ducharme, 1989; Noel, 
1988; Conseil des Colleges, 1988). For those who entered science at John 
Abbott in the fall of 1986 Boisset, MacKenzie, and Sidorenko (1989) found that 
after four semesters 55% of students had persisted, 25% had transferred to 
other programs, and 21% had left the college. Many students do not feel that 
they have control over their learning environment in Cegep, and frequently 
blame their own failure on factors that they consider to be beyond their 
control (Dweck and Elliott, 1983). Even students who do persist and succeed 
frequently do not see themselves as controlling their own fate. Indeed 
Boisset, MacKenzie, and Sidorenko (1989) found that only about a third of the 
students in their sample felt they had much control over what happened to 
them. Their study did find a gender difference in the overall rate at which 
students transferred out of science in that there was a tendency for females to 
attribute their lack of success to intrinsic factors that they could control, 
whereas males tended to attribute their failure more to extrinsic factors. 
Davis and Steiger (1993) found that one of the most reported reasons for 
dropping out of science at Cegep was loneliness, compounded by the 
emphasis on silent listening to teacher talk and solitary work doing 
calculations. 

1.21. Approach To Studying And Learning 

While it is true that students can control the length of time and 
amount of effort they may devote to the learning task, students may adopt 
varying study approaches when faced with a particular task because thev enter 
college with motives and intentions that do not match the college's learning 
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environments: they may have different perceptions of the teacher, the task, 
and the college. 

An approach to learning describes the relation between the student and 
the learning he or she is doing. It has elements of the student's motivation 
and elements of the student's perceptions in it. For example, a student may 
learn in order to obtain the diploma he or she wants, or they may be 
interested in a topic and want to find out more; they may be seeking to avoid 
pressure from a parent or they may be seeking status or recognition. An 
approach to learning is not an ability a student has, it is a combination of 
motivation and strategy, and as such must be distinguished from a learning 
style which refers to how the student structures their learning activities, and 
is not sensitive to context (Schmeck, 1988). 

Distinctions have been made in the research literature between three 
approaches: surface, deep, and achieving (Biggs, 1993; Entwistle, Hanley, & 
Hounsell, 1979; Marton and Saljo, 1976, 1984; Ramsden, 1985). 

In the surface approach the motivation is external: school is a means to 
an end. Students adopting this approach need to balance avoiding failure 
against working too hard. The strategy is to focus on memorizing or applying 
procedures unreflectively, rote learning. 

In contrast, students who use the deep approach will be motivated by 
personal curiosity. They will read widely, discuss, reflect on the material, 
focus on relations between parts, the structure of the problem as a whole, and 
the application of theory to real problems (Ramsden, 1991). 

The achieving approach is somewhat of an amalgam of deep and 
surface . The student adopts deep or surface level strategies according to what 
he or she judges to be most efficient for obtaining grades. This approach is 
based on a particular form of extrinsic motive: the ego-enhancement that 



comes from visibly achieving (Biggs, 1987). The student allots time and effort 
to tasks, using study skills such as planning, being neat and self-disciplined, in 
accord with the perceived contribution of the task to a high grade. The 
relationships between the strategies and motives is conveniently displayed in 
a figure: Figure 1.1. 



i 



Approach 



Surface 



Deep 



Achieving 



Figure 1.1 
Approaches to Learning 

Motive 



Strategy 



Surface motive (SM). is Surface Strategy (SS) is repro- 

instrumental: main purpose ductive: limit target to bare 
is to meet requirements essentials and reproduce 



minimally: a balance 
between working too hard 
and failing 

Deep Motive(DM) is 
intrinsic: study to actualize 
interest and competence in 
particular subject 

Achieving Motive(AM) is 
based on competition and 
ego enhancement: obtain 
highest grades whether or 
not material is interesting 



through rote learning 



Deep Strategy(DS) is 
meaningful: read widely, 
connect with previous 
relevant knowledge. 

Achieving Strategy(AS) is 
based on organizing ones 
time and working space: 
behave as a model student. 



The approach adopted by a student is determined by the task and the 
student's perception of the situational demands of the teaching and testing 
(Ramsden, 1988). The learning that results from the difterent approaches has 
been shown to depend on the approach adopted, the academic motivation of 
the student, the student perception of the learning climate, and what the 
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^ examiner rewards (Biggs, 1978, 1989; Craik and Lockhart, 1972; Svensson, 

| 1977). 

Deep learners appear to perform as well as surface learners on low 
£ level, or surface, tasks, and do much better on higher level tasks. In physics 

Prosser and Millar (1989), in a study of approach-outcomes of first year 
students found a strong connection between a deep approach and the 
abandonment of an Aristotelian view of mechanics in favour of a Newtonian 
view. Students also differ substantially in their capacity to clearly identify the 
£ nature and substance of the task demands; they may be able to correctly 

identify the hidden demands of the task but be unable to adapt to its demands 
(Marton & Saljo, 1976). 

There have been different methodologies used in assessing a student's 
approach to learning. Marton and Saljo (1976) interviewed students as they 
£ undertook a reading task and subsequently described the qualitatively 

different learning outcomes using (and introducing) the terms deep and 
surface. Johansson, Marton, and Svensson, (1985), and Prosser and Millar 
(1989) interviewed students as they undertook a series of conceptual tasks in 
physics and from an analysis of the transcripts established the student's 
£ learning approach. The Study Process Questionnaire or SPQ was developed 

by Biggs (1987) to assess approach to learning of college and university 
students. Biggs (1993) has reviewed the development of the instrument and 
the ways in which psychometric techniques such as factor analysis, 
information processing theories, and eontextually based work on student 
| approaches to learning have contributed to the development and 

understanding of both the instrument and the constructs it utilizes. The 
£ questionnaire has been used bv Biggs and others (e.g. Beckwith, 1991; Biggs, 

j 1987; Hley, 1992; I Iegar'y-1 Ia/el & Prosper, 1991) in a variety of settings and 
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disciplines. The validity of the constructs of the instrument have been 
confirmed using populations of high school, community college, and 
university students in Australia, and while test-r^test reliability is not 
necessarily an appropriate index for an instrument that can be used to track 
changes in student characteristics, none-the-less stability has been 
demonstrated (Biggs, 1987). 



1.22 Prior Knowledge And Understanding Of The Concept Of Force 

Years of experience give every student in introductory physics a well- 
established system of commonsense beliefs about how the world works. 
Many previous studies have compared this prior knowledge with the views 
of expert physicists (Clement, 1982; Halloun and Hestenes, 1985; McDermott, 
1984; Viennot, 1979), and it has been established that students' commonsense 
views are frequently incompatible with Newtonian concepts, (hence the label 
"misconceptions"), and that conventional instruction does little to change 
these beliefs (Hake, 1994). The Force Concept Inventory (Hestenes, Wells & 
Swackhamer, 1992) was developed from the earlier Mechanics Diagnostic Test 
(Halloun and Hestenes, 1985) to determine students' understanding of the 
concept of force. The test-retest reliability and validity of both the Diagnostic 
and the Inventory have been established by the authors using populations of 
high school, college, and university students in settings as diverse as high 
school in Arizona and first year physics classes at Harvard. In Quebec 
Desautels (1985) and Dickie (1988) have shown that students at both a 
francophone (Rosemont) and an anglophone (John Abbott) Cegep have a 
poor understanding of the concepts of mechanics in agreement with the 
findings in other countries such as France, New Zealand, and the United 
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States (Halloun & Hestenes, 1985; Hestenes, Wells & Swackhamer, 1992; 
Osborne & Freyberg, 1985; Viennot, 1979). Hegarty-Hazel & Prosser (1991) 
have shown that prior knowledge affects the adoption of study strategies and 
influences post knowledge. The authors of the Inventory have remarked on 
the importance of the attitude and motivation of students in overcoming 
misconceptions about force, and also the tendency of teachers to avoid "hard" 
conceptual questions in favour of quantitative problems where the answer is 
a number obtained by substitution into a formula after some algebraic 
manipulation. 

1.23. Assessment In Physics 

At John Abbott College, as at many other colleges and universities, a 
typical physics course will have a final examination, one or two mid-term 
tests, and possibly a number of quizzes and assignments. Most institutions 
have available in their library files of old examinations which students 
consult when preparing for an exam. While the course syllabus provides an 
overview of a course most students will have a better understanding of the 
aims and methodologies of a teacher after the first test. It has long been 
accepted (Michels, Sears, Verbrugge and Palmer, 1957; Fowler, 1969) that 
physics teaching at the college level should accomplish two goals; 1) an 
understanding of a core of knowledge, and 2) the systematic development of 
the methodology of physics. The requirements for physics tests to achieve 
these aims have been reviewed by a number of authors who have proposed 
different schemes to ensure that both aims are met (Aubrecht, 1990; Aubrecht 
and Aubrecht, 1983; Ferris, I960; Fowler, 1969; Kruglak, 1%6). For example 
Kruglak (1966) has suggested the use of a check-off grid to ensure that a test 



contains items that test 1) the recall of facts, 2) the use of applications that use 
these facts, and 3) the ability to apply principles to novel situations. Over 
time, there has been a shift in thinking about the relationship between 
teacher and student. In the past, the teacher was considered to be the 
dispenser of knowledge while the student was the passive listener. This 
viewpoint has shifted to that of the teacher as the guide and the student as the 
decision maker and constructor of knc fledge (Gallagher, 1994; White, 1992).. 
This shift in viewpoint has changed the emphasis between process and 
content but not the need to assess both. 

1.24c Thinking Skills In Physics 

Research on student perceptions and motivation suggests that students 
do not focus on the content goals of school work but implicitly ask: What do I 
have to do? How do I have to do it? Can I do it? Do I want to do it? 
(Blumenfeld, Mergendoller, and Swarthout, 1987). What abilities and skills 
do physics students need to answer these questions for themselves? Do 
physics teachers see answering these questions as goals for their instruction? 
College science teachers identify a variety of objectives for their courses One 
study asked 200 physics professors at both two year and four year colleges to 
identify the abilities that they valued most in their students: they found that 
visualization, facility with mathematics, logic, and problem solving were 
most valued (Peltzer, 1988). On a more general level cognitive research on 
science teaching shows the importance of four different types of learning: 1) 
subject matter learning; 2) knowledge of principles, rules, and specific 
situational knowledge; 3) learning strategies: and 4) metacognitive learning 
(Witlrock, 1994). After interviews with students and professors at 
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universities in Quebec and elsewhere Donald (1992) has described the 
thinking skills valued and developed in higher education. The levels of her 
model are: description, selection, representation, inference, synthesis, and 
verification. Donald found that physics professors believed that these skills 
were developed by problem solving (Donald, 1993). 

Despite the beliefs of many physics professors in the appropriateness of 
problem solving as both a teaching and assessment methodology (Aubrecht, 
1990; Van Heuvelen, 1991), it has been pointed out by McDermott (1991) that 
there is no convincing evidence that reasoning ability improves as students 
work the standard problems in an introductory physics course. In the Cegeps, 
problem solving in the form of assignments, problem sessions, tests, quizzes, 
and final exams is predominant in physics teaching. Supporting the wide 
acceptance within the physics community of problem solving as both a 
teaching and an assessment methodology is a large body of literature 
concerned with identifying and comparing the ways in which students 
approach problem solving (Clement, 1982; Fuller, 1982; Heller and Re if, 1984; 
Larkin, 1983; Zajchowski and Martin, 1993), or comparing the performance of 
novices with that of experts (Champagne, Gunstone, & Klopfer, 1985; Chi, 
Feltovich & Gla&er, 1981; Larkin, 1983). Ferguson-Hessler and de Jong (1987) 
have pointed out that for successful problem solving in physics students 
must not only possess a knowledge structure made up of 1) subject matter 
knowledge (both declarative and procedural), 2) knowledge of strategy, and 3) 
knowledge of the problem situation, but must be able to access and apply this 
knowledge structure. Reif, Larkin, and Brackett (1976) in an analysis of what 
prerequisite abilities a student needed in order to use a relation for solving 
physics problems defined "understanding a relation" in terms uf four 
categories ot abilities; 1) statement and example. 2) quantities in the relation 
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(vector/number, sign, units, ... ), 3) the relation itself (applicability, 
dependence, comparison, ... ), and 4) organization of the relation (when and 
where to apply). 

If problem solving is to be used to measure the thinking skills and 
degree of understanding of physics principles achieved by students it must be 
recognized that students generally approach problems according to their 
surface features, such as the presence of an inclined plane, rather than the 
underlying physical principles, such as the conservation of momentum or 
energy (Chi, Feltovich and Glaser, 1981). While it has been shown that 
problems that are unfamiliar to the solver but are structurally similar to 
familiar ones can be used to measure understanding (Gagne and White, 1978; 
Mayer, 1974), when physics test questions that cannot be solved by superficial 
application of a problem solving algorithm are asked, performance drops 
significantly (Halloun and Hestenes, 1985). 

1.25. Classifying The Intellectual Demands Of Learning Tasks In 
Physics 



The intellectual demands of a learning task in physics are defined by 
the answers students are required to produce and the routes that can be used 
to attain these answers (Doyle, 1983). In undertaking an analysis of the 
intellectual demands of the learning tasks, one is not looking at the physics 
content that is being asked for but at the behaviours and processes that are 
being required. This breakdown is necessary in order to apply the constructs 
of psychology to the demands of the tasks (Resnick, N76). 

In an exploratory study, Klatt (1991) hypothesized that because many 
physics problems requiu the student to construct a diagram from a verbal 
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description, the student is engaged in visualization and geometrical thought. 
His study analyzed the implicit geometrical content of problems from the 
kinematics and dynamics units of the Ontario Academic Course program 
using the classification scheme of van Hiele (1986). 

Another way to conceptualize the reasoning skills that students bring 
to bear on a problem is their information processing ability; that is the limit of 
the number of different ideas or schemes students can hold in their working 
memory at one time. Paseual-Leone defined this limit as the usable M-space 
(Paseual-Leone, Goodman, Ammon, & Subelman, 1978), and the demand of a 
problem as the M-demand. Niaz (1993a, 1993b) has reviewed the role of Neo- 
Piagetian theory and the effects of M-space and M-demand on problem 
solving in science. In chemistry Johnstone and El-Banna (1986) have pointed 
out the usefulness of mental capacity, or M-space, as a predictor of 
performance in chemistry exams, while Niaz has examined the relation 
between the M-demand of chemistry problems, the M-capacity of students, 
and student performance on solving chemistry problems (Niaz, 1987, 1988). 
In physics Roth (1987, 1990, 1991) has examined the influence of M-space on 
the amount of practice needed to induce problem solving strategies in 
physics. 

Aubrecht (1990) used a different classification scheme of the intellectual 
demands of physics exams based on three broad categories of logical processes 
(recall, interpret, apply), while Ferris (I960) used a scheme based on Bloom's 
taxonomy (Bloom 195')). In an analysis of first year final exams in a number 
of different subjects, including physics, Crooks and Collins (1986) used three 
categories, 1) straightforward recall, 2) straightforward application of formulae 
or principles; comprehension, and 3) analysis, synthesis, evaluation, solution 
of novel problems. 
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1.26 Taxonomies Of Intellectual Demands 



There have been a number of descriptive frameworks offered to classify 
and identify the intellectual demands of objectives and/or assessment items 
(Biggs, 1991; Donald, 1985; Gagne, 1977; Merrill and Tennyson, 1977). 
However the most widely used classification scheme is that developed some 
forty years ago at the University of Chicago by a team under the editorship of 
Benjamin Bloom (1956). The "Taxonomy of Educational Objectives, The 
Classification of Educational goals, Handbook I: Cognitive Domain" or 
"Blooms Taxonomy" as it is widely called, was developed in part to assess the 
demands of course objectives, and in part to examine the demands of 
examinations. It was developed in a pragmatic fashion from the ideas of 
several working groups and has come to be widely accepted. The six main 
levels of the taxonomy, which are hierarchical, are 1) Knowledge - recalling 
information much as it was learned, 2) Comprehension - reporting 
information in a way other than how it was learned to show that it has been 
understood, 3) Application - use of learned information to solve a problem, 4) 
Analysis - taking learned information apart, 5) Synthesis - creating something 
new based on some criteria, and 6) Evaluation - use of criteria in judging the 
value of something for a particular task or program. 

As part of a recent "Forty year retrospective" Krathwohl (1994), one of 
the original editors, suggested that the framework of the taxonomy has been 
most useful as it has been modified to better fit the discipline and the purpose 
to which it has been being applied. While no single taxonomy can be 
expected to apply equally well to a variety of disciplines because of the diverse 
nature of learning in different disciplines (Furst 1981), Bloom's taxonomy is 
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widely accepted in the literature, and has been applied to many disciplines in 
numerous cultures. In the Cegeps Bateman (1992) used the taxonomy to 
examine the intellectual demands posed by some 2300 assessment items taken 
from 27 social science courses of six Cegeps. 

Since problem solving is so important to physics, it is pertinent to 
review how Bloom treats problem solving. Problem solving can be classified 
at either the Comprehension or the Application level. The distinction 
between the two classifications is predicated on the assumption "If a student 
really comprehends something they can use it." Comprehension as Bloom 
uses the term implies a student can use an abstraction (formula, principle) 
when its use is specified. Application requires that a student correctly use an 
abstraction when no mode of solution is specified and the student must 
deduce the appropriate abstraction from the context of the problem. The 
process is illustrated by the chart of Figure 1.2 (Bloom, 1956, p. 121). 
Confounding this classification of solving a problem into either 
comprehension or application is the question of rehearsal. The objective in 
Application is to embody the idea of transfer of training; i.e. the application 
in a new situation of what has been learned in a different area (p. 123). Is the 
problem of a familiar type? If the problem is not novel, if the student 
"merely has to recall the situation in which he [or she] learned the 
abstraction" (p. 125) the item is at the Comprehension level. 

McGuire (1963), in modifying Bloom's taxonomy to better fit 
assessment in medicine, replaced Comprehension, Application, and Analysis 
with Generalization, Problem Solving of a routine type, and Problem Solving 
of an unfamiliar type. Simple interpretation of data was included with 
problem solving of a familiar type, while data analysis was included with 
problem solving. She also inverted the order of Evaluation and Synthesis. 

is id 



In a review of the intrinsic nature of academic work as it is experienced 
by students in the classroom, and the ways in which the intellectual demands 
of that work are related to academic success (Doyle, 1983), identified four 
general types of academic tasks; 1) memory tasks where the student is 
expected to reproduce information previously encountered, 2) procedural 
tasks, where the student is expected to apply a predictable formula or 
algorithm, 3) comprehension tasks, where the student is expected to recognize 
transformed information, draw inferences, and apply procedures to new 
problems, and 4) opinion tasks where students are expected to state a 
preference. He was interested in the actions of students after they were 
presented with a task, i.e. what the tasks lead the student to do, and the ways 
in which the tasks lead to learning. The classification was based upon the 
general categories of cognitive operations that are involved in task 
accomplishment (Greeno, 1976). 

The relevance of Doyle's work to classifying physics exam questions is 
that it explicitly adds consideration of the procedural complexity of a task to 
that of cognitive complexity. He explicitly recognizes the routine nature of 
applying a memorized formula by defining Procedural Tasks as those 
requiring students to apply a standardized and predictable formula or 
algorithm to generate answers. He contrasts Procedural and Comprehension 
tasks by pointing out the distinction between knowing how to apply an 
algorithm and knowing why the algorithm works and when it should be 
used. 




16 



Figure 1.2 

The problem-solving process in answering questions in the "Application" 

category. (Bloom, 1956, p. 121) 
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Recently Lawrence at the University of Michigan has integrated and 
refined the taxonomies of Bloom and Doyle to arrive at a hierarchy that 
differs from both (Lawrence, Hart, Kingan, and Campbell, 1994). Her group 
was lead to develop the "Michigan" scheme as a part of a project to examine 
the equivalencies of courses given at two year colleges to those given at four 
year colleges. In assessing the demands and equivalencies of courses at 
different institutions, this work stresses the influence of all aspects of the 
academic task both in the classroom and outside it. and the importance given 
by the instructor to different tasks and approaches. The taxonomy has 
different forms for different disciplines. The Michigan scheme for Calculus 
recognizes the rote or algorithmic nature of routine formula substitutions 
and it further recognizes that some problems require students to make 
choices about which rule or formula to apply based on the information given 
in the problem. These decisions are a part of the Comprehension levels of 
Bloom and of Doyle. In inserting a new level, Procedural /Comprehension, 
before Comprehension, the problem solving process is further split off from 
Comprehension and placed before it. This is in agreement with discussions 
held with physics teachers at a number of Cegeps, and with others familiar 
with both the use of Bloom's taxonomy and the nature of the learning task in 
physics. There remains the problem of where to place novel problems that 
require a student to apply known methods to unfamiliar situations. 



1.3 Purpose Of The Study 



This was an exploratory st.idy designed to determine the intellectual 
demands of Cegep physics exams. As such while it was predicted that most 
exams would be of a problem solving nature, no predictions were made as to 
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■ the cognitive level of the exam items. However previous work on school 

■ science textbook tests and standardized tests has shown that more than 75% of 
the items tested recall of facts and routine applications (Garcia and Pearson, 

I 1994). Similarly in her study of assessment in the Social Science programs of 

the English language cegeps, Bateman (1992) found that more than 80% of 
m assessment items were at the Knowledge or Comprehension levels. Crooks 

■ and Collins (1986) in a study of the skills first year university students were 
required to demonstrate in the final examinations of twelve different 

■ subjects, including physics, found that the most common level was that 
requiring the straightforward application of formulae together with 

m comprehension as defined by Bloom. 

m The second part of the study was to determine the approach to learning 

of students studying physics and to explore relationships between the 

■ student's approach and their learning outcomes as measured by grades in the 
courses and exams, and the cognitive demands of those exams. 

m 1.4 Significance Of The Study 

■ Not just in Quebec, but worldwide, the learning outcomes expected of 
colleges are being changed from specifying content to specifying the 

| development of abilities. The importance of seeing the connection between 

_ assessment and the kinds of learning that students, achieve is not a new issue 

but it has become more important with the current emphasis on 

■ accountability, both financial and intellectual. The learning outcomes are 
determined by the tasks and activities students undertake. 

| If assessment encourages the memorization and recall of isolated pieces 

_ of information, students will be ill-equipped to adapt to new needs 

eric 



throughout their lifetimes because of the rapid obsolescence of factual 
information. It follows that the knowledge that students accumulate during 
schooling is less important than the learning skills and habits they develop. 
If assessment encourages meaningful application and understanding of 
methods and principles, and both teachers and students share an 
understanding of the outcomes of the teaching learning/process (Solas, 1992), 
students will be better able to cope with the changing demands and challenges 
they will encounter. The role of exams in directing and shaping the work of 
both students and teachers is pivotal: not only can the examinations 
encourage meaningful learning but they can foster teaching that rewards 
appropriate learning. 
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CHAPTER 2 



Methodology 

2.0 Introduction 

This study is in two parts: 1) the learning approach of physics students, 
and 2) the cognitive demands of exams in physics. This chapter describes the 
subjects and measures used in the first part of the study then describes the 
development of the taxonomy used in the second part. 

2.1 Sample 

The first sample for the study consisted of 107 first semester students 
who completed the measures detailed below at the start of the year, followed 
Mechanics 101 and wrote the final exam. The second sample consisted of the 
35 students who, after passing Mechanics 101, entered Electricity and 
Magnetism 201, wrote the final exam in the course, and repeated the 
measures at the end of the year. 
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2.2 Measures 

2.21 Study Process Questionnaire 

The Study Process Questionnaire (SPQ) is a 42-item group administered 
instrument. Each item consists of an affirmative self-report statement that 
describes a student's strategy or motive. An example of a motive statement is 
" I find that at times studying gives me a sense of deep personal satisfaction/' 
An example of a strategy statement is "I summarize suggested readings and 
include these as part of my notes on a topic/' After consultations with 
students and teachers the wording of the questionnaire was changed to better 
conform to common usage in the Cegeps (e.g. tertiary to post secondary, 
lecturer to teacher, rote to learn by heart, ... ). For each item of the 
questionnaire the student responds on a five point Likert scale: 1 - this item is 
never or only rarely true of me, 2 - this item is sometimes true of me, 3 - this 
item is true for me about half the time, 4 - this item is frequently true of me, 
and 5 - this item is always or almost always true of me. Seven items of the 
questionnaire are constructed to reflect each of the subscales: surface, deep, 
and achieving motivation, and seven items to reflect each of surface, deep, 
and achieving strategy. The subscales are further combined to give the scale 
scores; Deep Approach, obtained by adding the two deep sub-scale scores 
(DM+DS), and similarly the Surface Approach (SM+SS), and Achieving 
Approach (AM+AS), scale scores. The relations between these scores is 
displayed in Figure 2.1. 
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Figure 2.1 
SPQ Scale and Sub-scale scores 
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The scale scores were used as dependent variables to identify a student's 
approach to learning, and the sub-scale scores were used as independent 
variables to assess changes in the motives and strategies over the one year 
period of the project. A student's approach to learning was determined by 
combining the scale scores. A score below the thirtieth percentile was 
considered below average, while a score above the seventieth percentile was 
considered above average. Norms are available for male and female students 
in different faculties of universities and colleges of applied arts and sciences in 
Australia. Students in this study were in the first year of the pre-university 
science stream at Cegep, therefore results were analyzed using Biggs's norms 
for science university students. 



2.22 The Force Concept Inventory 

The Force Concept Inventory is a twenty nine item multiple choice 
questionnaire thai was used to determine a students prior knowledge of the 
concept of force and the agreement between the student's understanding and 
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the Newtonian understanding (Hestenes, Wells, and Swackhamer, 1992). The 
score on the inventory was used in this study as a dependent variable. The 
questionnaire was administered at the start of the study to measure the 
students prior knowledge, and again one year later at the end of the study to 
those students who had continued in physics to measure the change in their 
understanding. Because the score on the test is a measure of the student's 
understanding of the concept of force rather than of their ability to apply 
formulae, connections between the student's score and the student's approach 
to learning were also sought. 

2.23 Cote Finale 

The Cote Finale (science) is a weighted average of a student's high 
school grades for Secondary IV and Secondary V. It is calculated for students 
who took high school in Quebec, is used in determining admission to the 
college. The cote finale was used as a measure of high school performance. 

2.3 Coding Scheme For Assessment Items 

Bloom's taxonomy has six hierarchical levels, knowledge, 
comprehension, application, analysis, synthesis, and evaluation. Preliminary 
work showed. that the examinations and other assessment items of this study 
involved the three lowest levels of Bloom's taxonom) but that the definitions 
of these levels did not adequately represent the intellectual demands of 
assessment items in physics. Given the predominance of problem solving as 
both a teaching and an assessment methodology in physics, it was necessary to 
consider the place of problem solving in the taxonomy and in particular two 
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linked issues; novelty versus rehearsal, and rote application of an algorithm 
versus understanding. The issue of whether a problem was novel was 
addressed by examining quizzes and tests which confirmed the initial 
hypothesis that almost all problem types were rehearsed. Indeed one could 
ask; Should a novel problem be part of a final exam in physics? The 
consensus of a number of physics teachers, at both John Abbott ?nd other 
colleges, was that no, the final exam was not the place. The second issue was 
rote application versus understanding. When a student was applying a 
problem solving algorithm to the solution of a typical or "text-book type" 
problem, was the student reflecting on each step and understanding why 
choices were being made or were they just following a well worn path? A 
path that the student had seen demonstrated in class or had rehearsed as 
assignments were completed. The consensus of discussions with teachers was 
that it was generally the latter. Accordingly, it was decided to adopt the point 
of view of Doyle (1983), and of Lawrence et al (1994) and place Comprehension 
after routine problem solving. This is some-what at odds with Bloom's 
statement "If a student really comprehends something he can apply it" (1956, 
p. 120) because it accepts that the converse of this statement is not necessarily 
true. Deciding whether a student has understood a procedure could be 
resolved by interviewing the student, but examination of their correct written 
answer to a typical exam question is unlikely to reveal whether they were 
following an algorithm, or had an understanding of why the procedure was 
appropriate and successful. 

The levels of the taxonomy used to analyze questions are given below. 
More complete definitions are given in Appendix 1. 
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Memory: Recalling information much as it was learned. 

Procedural/Algorithmic: Following a routine series of steps in solving a 
problem. The problem is familiar and the rule or formula 
is either given in the problem or very familiar from 
previous rehearsal. 

Procedural/Comprehension: Solving a problem that requires that 
choices be made about the which rule or formula to apply 
based on the information given in the problem. 

Comprehension: Recognize transformed or paraphrased information. 

Draw inferences from previously encountered 
information. In applying a rule or formula demonstrate 
understanding of when, why, and how the relation can be 
applied. 

The first category, Memory, demands the recognition or reproduction 
of information previously encountered. Bloom considers that such tasks do 
not require thinking and distinguishes this level from other intellectual tasks, 
tasks that require some content to act on. For example one does not just 
think, one thinks about projectile motion. 

In physics there are many short, routine problems requiring little 
thinking or understanding. Bloom places such operations as part of 
knowledge but they ai . so common in physics that a separate category, 
Procedural/Algorithmic, is warranted. Similar distinctions in different 
disciplines have been made by Doyle (1983), Lawrence et al (1994), and 
McGuire (1963). 

In the third category, Procedural/Comprehension, the student has to 
make choices, has to make decisions and judgments about what procedure to 
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follow, may even have to carry out some analysis or form an opinion - but at 
a straightforward level. The problem is not novel. In discussions with Cegep 
physics teachers it was agreed that placing this level before Comprehension 
was appropriate and proper, a decision also made by Lawrence et al, (1994). 

The label Comprehension is used by Bloom, by Doyle, and by Lawrence 
to describe similar but slightly different levels of understanding. Bloom talks 
about transforming information to demonstrate that it has been understood 
and about applying a formula when its use is specified. Doyle defines it in 
terms of recognizing transformed information, and also in terms of choosing 
between several procedures in solving a problem. Lawrence talks about 
understanding the "gist" of a problem, the how and why procedures are used. 
In the present work the four categories of abilities defined by Reif, Larkin and 
Brackett (1976) as constituting understanding of a relation are used as the basis 
for the working definition of comprehension. These include the 
transformation abilities of Bloom or Doyle, for example translating from a 
table of values to a graph, or interpreting and using the information given in 
a graph. The understanding of why a problem solving procedure works 
places this after procedural/comprehension. 

In adopting these levels, the cognitive demands of problems have been 
split into four levels: memory, rote application, those requiring limited 
comprehension, and those requiring understanding of principles rather than 
just the demonstration that an algorithm can be applied. 
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2 A Procedures 

2.41 Selection Of Sample. 

The subjects for the study were science students taking Mechanics 101 
in the fall 1993 semester at John Abbott College. The Study Process 
Questionnaire anci the Force Concept Inventory were administered in six 
classes during the first two weeks of classes. There were both first and second 
year students in these classes because the fall semester of 1993 v r as one of 
transition to the "new" science program for the anglophone colleges. Of the 
267 students who completed the SPQ, 153 were first year students, 88 were in 
their second year, and the remaining 26 were mature students. The Force 
Concept Inventory was administered in the second week of classes. 
Inevitably, there were students who wrote the first instrument but not the 
second and vice versa. The Cote Final (science) for the first semester students 
was obtained from the college. 

Complete data was obtained for 111 students. At the end of the 
semester 107 of the students wrote the physics final exam. These 107 students 
constituted the main cohort in this study. 

Of the 83 students who passed Mechanics 101, 52 enrolled in Electricity 
and Magnetism 201 and at the end of the second semester the 46 students who 
were still following the course were invited by letter and telephone to re-write 
the SPQ and the FCI. A $5.00 honorarium was paid upon completion of the 
questionnaires: Thirty-six accepted the invitation (78%), but one student did 
not subsequently write the 201 final exam. These 35 students constituted the 
second cohort in this study. 




2.42 Determination Of The Approach To Learning 



Using the norms provided by Biggs (1987) for university science 
students the scale scores were designated as above average if they were above 
the seventieth percentile, average if they were in the percentile range 31 to 70, 
and below average if they were below the thirtieth percentile. Following 
Biggs, a student was classified as adopting a predominantly Surface Approach 
if their surface scale score profile was greater than both of their deep and 
achieving profiles. In an analogous manner students adopting 
predominantly Deep or Achitving Approaches were identified. Biggs also 
considers two combined approaches, Surface Achieving and Deep Achieving. 
A student who had equal profiles for surface and achieving, and a lower 
profile for deep approach was classified as Surface-Achieving, while a student 
with equal deep and achieving profiles and a lower surface profile was 
classified as Deep-Achieving. In addition there were many students for whom 
a predominant approach did not emerge in that they were equal in deep and 
surface approaches and weaker in achieving. 

2.43 Selection Of Quizzes, Tests, And Final Exams 

The primary data for this part of the study were the final physics 
examinations Mechanics 101, Electricity and Magnetism 201, and Waves and 
Optics 301, given in December 1993 and May 1994 at John Abbott College 
together with a set of quizzes and tests for each course. Given that only a 
limited number of exams were examined it was decided to determine 
whether the assessment practices at John Abbott were representative. The 
final exams of physics 101, 201, and 301 given in December of 1993 were 
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obtained from five anglophone Cegeps. In the case of courses that did not 
give a final exam but used the grades from the term tests, these were obtained. 
Three of the colleges had a common marking scheme and common final 
exams. In the other two colleges teachers were free to set their own exam and 
grading policies. Structured discussions with teachers from the colleges 
showed that teachers exchanged tests and other assessments, and some 
compared average test marks and pass/fail rates to ensure uniformity across 
different sections of a course. At John Abbott the teachers of a course marked 
the common final exam collectively; i.e. one teacher would mark the first 
question in all scripts, another the second and so on. At another college a set 
of exams had been independently marked by four teachers and the grades 
assigned compared. In at least one college the same assignments were handed 
out to all students, and solutions posted in the hallways and were available in 
the library. The researcher and the two coders (who were experienced physics 
teachers from colleges other than John Abbott) reviewed all the exams and 
other assessment items and it was agreed that coding the assessments from 
John Abbott and the final exams /tests from two other colleges would give a 
representative picture of the practices at the five anglophone colleges. The 
assessments from the two colleges were selected because one had a policy of 
common final exams while one did not, and because the coders came from 
these colleges and could provide input as to the novelty or otherwise of the 
questions of the assessments. In all thirty quizzes, fifteen term tests, and ten 
final examinations totalling 710 items were coded. 



2.44 Coding Of Items Of The Quizzes, Tests, And Final Exams 



The process of assigning a final code to each question or part question 
(710 items) followed a number of steps. First, solutions to each question were 
prepared by the researcher. The majority of these solutions were annotated 
with the thinking processes that were followed. In making these annotations 
the researcher acted as an expert physicist trying to think like a novice. The 
construction of these solutions contributed to the development and 
understanding of the coding scheme. Second, after a training session, the 
coders were provided with a detailed rational for the coding scheme together 
with examples and two final exams were coded. The team then met to discuss 
the scheme and to compare codes. The coding scheme was adjusted and 
finalized. Then the coders and the researcher coded all 710 items 
independently. 

The final codes were assigned as follows. 

If the two coders agreed on the code this became the final code. The 
rate of agreement was 72%. 

If there was no initial consensus the code assigned by the researcher 
was considered. If it agreed with one of the codes assigned by the coders this 
became the final code. This was the case for all but 16 cases. The remaining 16 
items were discussed and final codes assigned. The chief cause of 
disagreement was between Memory and Comprehension. For example; was 
an electric field diagram, or the derivation of a simple harmonic motion 
formula, remembered or understood? In addition 5 items on which the 
coders agreed, but which one or the other had indicated they wanted to 
discuss, were also reconsidered. 
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In addition to being coded for intellectual demands each item was also 
classified according to the six different activities that the student had to 
undertake in answering the question. Each item was classified, 1) as a 
problem, 2) as requiring the construction or interpretation of a diagram, 3) as 
requiring the construction or interpretation of a graph, 4) as requiring a 
written response, 5) as requiring a derivation or proof, or 6) as being a 
multiple choice question. 

2,5 Analysis Of The Data 

The FCI and the SPQ responses were scored using Op-Scan sheets and 
the college computer system. In the case of the SPQ the raw data was 
transferred to a personal computer, and the sub-scale and scale scores 
computed with the help of a spreadsheet program (Excel). In exploring the 
relationships between the performance of the students on the final 
examinations and the measures, a variety of statistical tests were applied as 
appropriate; Anova, t-test, Tukey method of multiple comparisons, Pearson 
product-moment coefficients of correlation, and linear regression analysis 
(Glass and Hopkins, 1984). The statistical analysis of the results was done 
using SYSTAT. 
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CHAPTER 3 



■ Approach To Learning 

3.0 Introduction 

This chapter describes the results of using the Study Process Questionnaire 
I to identify the approach to learning of the students, and the changes in the sub- 

■ scale and scale scores over the two semesters of this study. 

I 3.1 Approach To Learning Of Incoming Students 

I Within the cohort of 107 students who wrote the Study Process 

m Questionnaire at the beginning of their first semester at the college, according to 

Biggs' norms 12% were classified as adopting a predominantly Surface Approach 

■ to learning, 5.6% Deep, 14% Achieving, 29% Surface Achieving, 12% Deep 
Achieving, and for 27% the approach was not identified because they were 

| equally strong in both surface and deep and lower in an achieving approach, 

a Results are shown in Table 3.1. 

The results indicate that most incoming students show a Surface or a 
I Surface Achieving Approach. The students wrote the instrument in their 

physics class, and were told to think about their physics courses when answering 
| the questions. These students approach physics with the intention of 
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memorizing formulae rather than understanding concepts, and are prepared to 
modify their approach to do what they think will gain them marks. 



Table 3.1 

Percentage Of Incoming Students Classified As Adopting A Particular Learning 

Approach (n = 107) 



Surface Deep Achieving Surface- Deep- Un- 

Achieving Achieving determined 

12 5.6 14 29 12 27 



3.2 Approach To Learning After Two Semesters Of Instruction 



Sixty three percent of the students who passed Mechanics 101 proceeded 
directly to Electricity and Magnetism 201. For the sample of these academically- 
on-track students who re-wrote the measures at the end of the second semester 
(35 or 78% of those who continued), there were increases in the percentages of 
students classified as adopting both a Surface and a Deep Approach, and declines 
in those adopting both an Achieving Approach, and the composite Deep- 
Achieving and Surface-Achieving Approaches. The findings show that after two 
semesters at Cegep the percentage of students classified as adopting a Deep 
Approach according to Biggs's norms has increased from 5% to 20%. Should we 
be satisfied when, after two semesters of physics, only 20%» report that they try to 
understand the concepts of physics? The percentages of students classified as 
adopting each approach when the SPQ was written at the start of the first 
semester (pre), and at the end of the second semester (post) are shown in Table 
3.2. 
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Table 3.2 

Percentage Of Students Classified As Adopting A Particular Learning Approach: 
Cohort Of 35 Who Rewrote The SPQ At The End Of Their Second Semester 





Surface 


Deep 


Achieving 


Surface 
Achieving 


Deep 
Achieving 


Un- 
determined 


Pre 


5.7 


2.8 


31.4 


28.6 


17.1 


14.3 


Post 


14.3 


20 


2.8 


11.4 


14.3 


37 



3.3 Changes Over Two Semesters Of The SPQ Sub-Scale And Scale Scores 

While the increase in those classified as adopting a Deep Approach and 
the decrease in those adopting an Achieving Approach is encouraging, the 
changes were only partially supported by changes in the sub-scale and scale scores 
of the SPQ. Group means were compared using the pairwise t-test and after the 
two semesters of instruction the Surface Motivation had increased, there was no 
change in the Deep Approach, Achieving Approach had decreased, and there 
were trends that point to a decrease in Deep and Achieving Motivations and 
Achieving Strategy. The results are presented in Appendix 2. 



CHAPTER 4 



Intellectual Demands Of Quizzes, Tests, And Final 

Examinations 

4.0 Introduction 

This chapter describes the results of the coding of the items of the 
quizzes, tests, and final examinations according to the taxonomy 
developed in this study. 

4.1 Level Of Thinking Of The Assessment Items 

The coding scheme developed classifies the level of thinking 
required by the assessment items into four hierarchical levels; Memory, 
for items remembered rather than solved; Procedural/Algorithmic, for 
problems of a routine nature; Procedural/Comprehension, for problems 
and items requiring limited comprehension or understanding; and 
Comprehension, for problems and items requiring understanding of 
principles and concepts, and non-routine translation from one 
representation to another such as from a table to a graph. No items were 
encountered at the higher levels of thinking: analysis, synthesis, or 
evaluation. While problem solving involves elements of each of these 
abilities, the conclusion of the researcher and the coders was that few of 
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the items were novel, and although many required considerable 
mathematical manipulation, the thinking demanded fell short of that 
required by the higher levels of Bloom's taxonomy. Of the 710 items from 
the thirty quizzes, fifteen term tests, and ten final exams, most (58%) were 
at the Procedural/Comprehension level, while 28.5% were at the 
Comprehension level, 12.7% were at the Procedural/Algorithmic level, 
and 0.8% of the items were coded at the Memory level, as shown in Figure 
4.1. 
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Figure 4.1 

Percent Total Items By Level Of Thinking 



4.2 Categories Of Assessment Items: Problem, Diagram, Graph, Written 
Response, Multiple Choice, Theory 

The majority of the 710 items examined (75.9%) were problems. 
Just over ten percent required the construction or interpretation of a 
diagram; 6.7% required the construction or interpretation of a graph; 4.5% 
required a written description or explanation. There were 2.1% multiple 

hi 
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choice questions, and 0.4% of the items required the student to 
demonstrate a proof or develop a theoretical expression. The percentage 
of total items accounted for by each category is presented in Figure 4.2. 
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Figure 4.2 
Percent Total Items By Category. 
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43 Level Of Thinking For Each Category 



Most items were problems and these ranged from the most routine 
to the complex and mathematically challenging. Rehearsal tended to 
reduce the level at which a problem might otherwise have been coded. In 
fact two problems, both from course 301, and from two different colleges, 
were coded at the memory level because the coders considered that while 
it would be possible to "solve" the problem, students would be more likely 
to remember the answer. Of the 539 items classified as problems most 
(69.8%) were coded at the Procedural/Comprehension level, 15% at the 
Procedural/Algorithmic level, 14.87o at the Comprehension level, and 
0.4% were coded at the Memory level, as shown in Figure 4.3. 
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Figure 4.3 

Percent Of Items Classified As Problems By Level Of Thinking 



There were 73 items classified as diagrams and 48 items classified as 
graphs. For the 73 items classified as diagrams, 80% were coded at the 
Comprehension level, 16.4% at the Procedural/Comprehension level and 
1.3% at each of the Procedural/ Algorithmic and Memory levels. For the 
48 items classified as "graphs," 37.5% were coded at the Comprehension 
level, 47.9% at the Procedural/Comprehension level, and 14.5% at the 
Procedural/Algorithmic level, as shown in Figure 4.4. 

Thirty of the 32 items classified as requiring a written response were 
coded at the Comprehension level and two at the Memory level. The 
multiple choice items were found in one of the mechanics final exams 
and were based on questions from the Force Concept Inventory of 
Hestenes, Wells and Swackhamer (1992); fourteen of the 15 items were 
coded at the Comprehension level and one at the Memory level. 
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Figure 4. 1 

Percent Of Items Classified As Diagram And As Graph By Level Of Thinking 



One of the three theory items was coded at the memory level and 
two at the Comprehension level. These three items were amongst the 
most difficult to obtain a consensus on: Were they remembered in a rote 
fashion, or understood? To what degree had they been rehearsed? 

4.4 Level Of Thinking Of Items For Courses 101, 201, And 301 

In all, 253 items of the course Mechanics 101 were coded, 236 items 
of Electricity and Magnetism 201, and 221 items of Waves and Optics 301. 
It was found that the percentage of items coded at the Comprehension 
level increased from 22.9% in course 101 to 30% in course 201 to 33% in 
course 301, and the percentage of items at the Procedural/Comprehension 
level underwent a corresponding decline from 64.8 to 57 to 51.5% 
respectively. The percentage of items coded at each level of thinking for 
each course is shown in Figure 4.5, and the complete data is given in 
Appendix 3. 
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While there is an increase in the percentage of items at the 
Comprehension level from course 101 to 201 to 301, and a corresponding 
decrease in items at the Procedural/Comprehension level, it should be 
remembered that no items were judged to require analysis, synthesis, or 
evaluation. 




Level of thinking 

Figure 4.5 

Percent Of Items By Course And Level Of Thinking 



4.5 Level Of Thinking By Grade 

To determine whether analyzing the level of thinking by number of 
items, rather than by grade assigned to each item, skewed the results, the 
grade assigned each item of the quizzes, tests, and final exams from John 
Abbott College for the winter semester 1994, was determined. It was found 
that in general teachers gave more weight to items coded at the higher 
levels of thinking and less weight to items coded at the lower levels. 
Results for the course Mechanics 101 showed that for those items coded at 
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the lower level of problem solving, Procedural /Algorithmic, the teachers 
grading the items assigned less weight, while those coded at the 
Procedural/Comprehension level were given more weight, and items 
coded at the highest level, Comprehension, were given slightly more 
weight as shown in Figure 4.6. This same pattern was found for courses 
201 and 301. The complete data is given in Appendix 4. 




Level of thinking 



Figure 4.6 

Course 101 John Abbott College Winter Semester 1994. Percentage Of Items At 
Each Level Of Thinking Compared With Percentage Of Grade Assigned To Each 

Level Of Thinking. 



4.6 Level Of Thinking By Grade: Quizzes, Tests, and Final 
Examinations 



The final grade in a course is made up of grades from the quizzes, 
| tests, final examination, and the laboratories. This study was concerned 

m with the thinking skills demanded by the quizzes, tests and final exams. 

• The interplay between the levels of thinking demanded by an item, the 
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grade assigned the item, and the assessment type (exam, test, quiz) of the 
item, can be looked at in two ways: firstly, in terms of the percentage of 
the grade of each type of assessment (quiz, test, final) coded at each of the 
levels of thinking; and secondly, in terms of the absolute contribution to 
the final grade of items coded at each level of thinking for each type of 
assessment. 

For the course Mechanics 101, it was found that the final exam 
contained a greater percentage of items coded at the Comprehension level 
than did the tests and quizzes, and a lower percentage of items at the 
Procedural/Algorithmic level: for all three types of assessment the 
greatest percentage of items were at the Procedural/Comprehension level, 
as shown in Figure 4.7. The complete data for all three courses is given in 
Appendix 5. 
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Figure 4.7 

Course 101: Percentage Of The Grade Of Each Type Of Assessment Due To Each 

Thinking Love' 
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For course Electricity and Magnetism 201 the percentages of each of 
the final exam, tests, and quizzes coded at each level were very similar: 
approximately 35% at the Comprehension level, just under 60% at the 
Procedural/Comprehension level and approximately 6% at the Procedural 
/Algorithmic level, as shown in Figure 4.8. 




Level of thinking 

Figure 4.8 

Course 201. Percentage Of The Grade Of Each Type Of Assessment Due To Each 

Thinking Level 
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For course Waves and Optics 301, the percentages of the final exam, 
tests, and quizzes at each of the levels of thinking was more varied than 
that of either 101 or 201, as shown in Figure 4.9. 

For each course the final exam contributed 40 marks to the total 
grade while the tests contributed 30 marks and the quizzes just 10 marks 
(the remaining 20 marks came from the laboratory reports). Because of the 
predominance of grades assigned to the Procedural/Comprehension level 
examination of the absolute contribution by type of assessment did not 
reveal significant patterns. 
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Figure 4.9 

Course 301: Percentage Of The Grade Of Each Type Of Assessment Due To Each 

Thinking Level 



Overall, the marks assigned to items at the Comprehension level 
increased from course 101 to 201 to 301, while the marks assigned to items 
at the Procedural/Comprehension level declined. At the lowest level of 
problem solving, Procedural/Algorithmic, more marks were assigned to 
items in course 101 than in either of the subsequent courses, and there was 
a very small number of marks given for memory items in courses 101 and 
301, as shown in Figure 4.10. 



ERLC 



46 




Level of thinking 

Figure 4.10 

Grade By Course By Level Of Thinking: John Abbott College Winter Semester 1994 



The increase in the grade due to items at the highest level of 
thinking encountered, Comprehension, as one goes from Mechanics 101 
to Electricity and magnetism 201, to Waves and Optics 301, would seem to 
be appropriate but we must remind ourselves that the higher levels of 
thinking, analysis, synthesis and evaluation were not found. 
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CHAPTER 5 



Approach To Learning, Intellectual Demands Of Assessment, And 

Performance 

5.0 Introduction 

This chapter reports the relationships between the approach to learning of 
the students as determined by the Study Process Questionnaire, the intellectual 
demands of the assessment tasks, and the performance of the students on the 
final exams and the Force Concept Inventory. 

5.1 Approach To Learning Related To Assessment Performance 

Significant links were found between the approach to learning adopted by 
students and their performances on both the Mechanics 101 and Electricity and 
Magnetism 201 final examinations. A step-wise linear regression of the scale 
scores of the SPQ, the Cote Final, and the Force Concept Inventory (the 
independent variables) with the Mechanics 101 final exam grade (the dependent 
variable), showed that high school performance, as measured by the Cote Final, 
was more strongly correlated to performance than the other measures and that 
both the FCI and Achieving Approach scores were positively correlated while the 
Deep Approach score had a negative correlation. Results are shown in Table 5.1. 
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Table 5.1 








Step-Wise Line 


ar Regression Mechanics 101 Exam Grade. 
Squared Multiple R = 0.948 
(n = 107) 




Variable 


Std Error 


Std Coef 


T 


P 


Cote Final 


U.l/D 


0.821 


5.65 




Force Concept 
Inventory 


0.608 


0.275 


3.67 


.000 


Deep Approach 


0.392 


-0.358 


-2.37 


.019 


Achieving 
Approach 


0.349 


0.244 


1.65 


.102 



High school performance was also more strongly correlated to 
performance on the Electricity and Magnetism exam than the other measures. A 
step-wise linear regression between the Cote Final, the FCI, and the scale scores of 
the SPQ (both measures being re-written at the end of the second semester) and 
the Electricity and Magnetism final exam grade showed that the Cote Final, the 
score on the FCI and the Achieving Approach score were positively correlated to 
performance, and Surface Approach score was nagatively correlated with 
performance. 

There was a clear relationship between intellectual demands of the exam, 
performance on the exam, and approach to learning. The fall 1993 Mechanics 101 
final exam had just 6.6% of its grade coming from items coded at the 
Comprehension level and there was a negative correlation between performance 
and Deep Approach. In contrast the winter 1994 Electricity and Magnetism exam 
had a much higher percentage, (34.8%) of its grade coming from items coded at 
the Comprehension level and there was a negative correlation between 
performance and Surface Approach. Results are shown in Table 5.2, 
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Table 5.2 

Step-Wise Linear Regression Electricity And Magnetism 201 Exam Grade 
And Measures Written At The End Of The Second Semester 
Squared Multiple R = 0.957 







(n = 35) 






Variable 


Std Error 


Std Coef 


T 


P 


Cote Final 


0.074 


0.985 


3.78 


.001 


Force Concept 
Inventory 


0.177 


0.313 


2.93 


.006 


Surface 
Approach 


0.142 


-0.697 


-2.84 


.008 


Achieving 
Approach! 


0.134 


0.382 


1.76 


.089 



■ There were differences between the approach to learning of students who 
passed Mechanics 101 and those who failed. A one way analysis of variance for 

I the initial sample showed that those who passed Mechanics 101 had a higher 

score on the Achieving Strategy sub-scale than those who failed. While there 

■ was no significant difference on the Achieving Motive sub-scale there was a 
m strong trend (p = .56) for those who passed to have a higher Achieving Approach 

score. This finding is in agreement with the finding that the majority of the 

■ marks (91.7%) on the fall 1993 Mechanics final exam came from items coded at 
the Procedural/Comprehension level, i.e. the problems were straightforward 

m and required only limited understanding of principles. The results are shown in 

m Table 5.3. 

There were also differences between the approach to learning of students 
I who passed and failed Electricity and Magnetism 201. Because the number (35) 

was small a statistical analysis was not attempted but those who adopted Deep or 

■ Deep Achieving approaches were more successful than those who adopted a 

© 51 , , 



Surface Approach. The students who participated in the second part of the study 
rewrote the SPQ during the last weeks of the semester shortly before the final 
examinations. Of the 35 students 22.8% (8) failed Electricty and Magnetism 201 
and 77% (27) passed. For the 14% (5) who were classified as adopting a Surface 
Approach when they re-wrote the SPQ three failed and two passed: of the 20% 
(7) classified as adopting a Deep Approach 6 passed and 1 failed, in addition all of 
the 18% (5) classified as adopting a Deep-Achieving Approach passed, while three 
of the four classified as adopting a Surface-Achieving Approach passed: the one 
student classified as Achieving passed. These findings show that a Deep 
Approach to learning was associated with success in the course, and can be 
related to the 28.4% of the grade in the course coming from items at the 
Comprehension level and 46.5% from items at the Procedural/Algorithmic 
level. 



Table 5.3 

Univariate Analysis Of SPQ Sub Scale Score Achieving Strategy And Scale 
Score Achieving Approach. 
(n = 107) 

Pass Fail 



SPQ sub-scale 
Achieving strategy 

SPQ Scale 



n 


= 93 


n 


= 14 






Moan 


G.D. 


Moan 


S.D. 


T 


P 


22.93 


4.80 


20.07 


5.38 


2.05 


.04 


48.68 


7.21 


44.36 


11.16 


1.94 


.056 



Pearson product-moment correlations were computed between students' 
grade on the Mechanics 101 final exam, and each of the scale and sub-scale scores 
of the SPQ. It was anticipated that there would be a positive relationship with 
deep and achieving scores and a negative relationship with surface scores. None 
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were found however. In addition there were no significant correlations between 
performance in the 101 final exam and the SPQ scales and sub-scales for the 
cohort of 35 students who, after successfully completing Mechanics 101, went on 
to follow Electricity and Magnetism 201. However there was a significant 
negative relationship (p = .51) between Surface Approach and grade in the 201 
final examination. The results are shown in Appendix 6. 

5.2 Approach To Learning And Score On The Force Concept Inventory 

The incoming students wrote the SPQ and the FCI at the start of their first 
semester. For these students, the ones who were classified as adopting a Deep 
Approach to learning scored significantly higher on the Force Concept Inventory 
than did the others. A Tukey HSD multiple comparison showed that the group 
classified as adopting a Deep Approach to learning scored significantly higher 
than those classified as adopting either a Surface (p = .012) or an Achieving (p = 
.001) Approach. In addition those classified as Deep scored significantly higher 
than those classified as either Deep Achieving (p = .005), or Surface Achieving (p 
= .003). None of the other between-groups differences were significant. All of 
the students classified as adopting a Surface Approach had FCI scores lower than 
that of the lowest score of the students classified as adopting a Deep Approach. 
Results are shown in Tables 5.4 and 5.5. 
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Table 5.4 






Means And Standard De 


viations For Score On Force Concept Inventory By 
Approach To Learning 






Approach 






Surface 
n=13 


Deep 
n=6 


Achieving 
n=15 


Surface 
Achieving 
n=31 


Deep 
Achieving 
n=13 


FCI (%) 46.4 
S.D. 10.9 


70.1 
10.8 


46.9 
13.0 


47.1 
14.2 


48.7 
13.8 



Table 5.5 

Analysis Of Variance For Approach To Learning And FCI Score 

df SS MS F p 

Between groups 5 349.5 69.90 4.08 .002 

Within-groups 101 1729.1 17.12 



The relationship between the Deep Approach and the score on the FCI is 
consist- nt with the aims of the two measures. A Deep Approach indicates an 
intention to apply principles to real-world problems and the FCI is known to test 
conceptual understanding of Newton's laws and the ability to apply the concepts 
to realistic situations. From Table 5.4 the mean score of the group of students 
identified as adopting a deep approach is 70%. The significance of this score can 
be gauged by the average pretest scores reported by Hestenes, Wells, and 
Swackhamer (1992) of between 34% and 52% for students entering university, 
and posttest averages of between 63% and 68%, and in the case of a posttest 
conducted in a class consisting mostly of physics majors at Harvard, 77%. The 
SPQ is identifying those incoming students who have a very good conceptual 
understanding of Newton's laws of motion. 
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5.3 Relationships Between Study Process Questionnaire Scale And Sub-Scale 
Scores, And Score On The Force Concept Inventory 

The FCI and the SPQ were written at the beginning and again at the end of 
the study. For both sets of data Fearson product-moment correlations were 
computed between students' scores on the FCI and each of the scales and sub- 
scales of the SPQ. There were significant negative relationships between FCI 
score and an Achieving Approach for the pretest results and between FCI score 
and both an Achieving Approach and a Surface Approach for the posttest results. 
The findings suggest that the students did not adapt their approach to the 
demands of the Force Concept Inventory (which requires conceptual 
understanding rather than rote application of Newton's laws). The results are 
given in Appendix 7. 

5.4 Changes In The SPQ Scale And Sub-Scale Scores And The FCI Score After 
Two Semesters Of Instruction 

For the cohort of students who re-wrote the measures at the end of the 
second semester, group means were compared using the pairwise t-test. Over the 
course of the two semesters the Surface Motivation increased. There were trends 
that point to a decrease in Deep and Achieving Motivation and Achieving 
Strategy, and a significant decrease in Achieving Approach. These findings 
when taken together with the finding that almost two-thirds of the marks in the 
final exams for both the Mechanics and the Electricity and Magnetism courses 
came from items coded at the routine levels of problem solving support the 
notion that the demands of assessment determine the type of learning. If the 
assessments do not encourage thinking students will adopt a surface approach. 
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The increase in the score on the FCI over the two semesters was 
significant: the increase of 12.3% was in agreement with increases reported by 
Hake (1994) for conventional instruction. Results are presented in Appendix 8. 



5.5 Relationship Between Performance And Score On The FCI And The Cote 
Final 

Pearson product-moment coefficients of correlation showed significant 
correlations between the performance on the Mechanics final exam and high 
school performance, as measured by the Cote Final, and prior knowledge, as 
measured by the FCI. For the sample which re- wrote the measures at the end of 
the second semester there were significant correlations between performance in 
the Electricity and Magnetism exam and the Cote Final, as well as substantial 
correlations between performance in the exam and the FC! scores obtained both 
when the measure was written on entry to the college and when it was re- 
written at the end of the second semester. The results are given in Appendix 9. 

5.6 Relationships Between Gender And Performance On The Final 
Examinations, The FCI And The SPQ 

Given the different persistence and success rates of male and female John 
Abbott science students reported by Boisset, MacKenzie and Sidorenko (1989) and 
the more recent study of Davis and Steiger (1993) on gender neutral instruction 
in Cegep physics courses, the results were examined for gender differences. The 
initial sample of 107 consisted of 38 females and 69 males; the follow-up sample 
of 35 students consisted of 13 females and 22 males. There were no significant 
differences in the scores of males and females on the scales and sub-scales of the 
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SPQ. Significant differences were found in the Cote Final, and the FCI scores. 
Initially the females in the sample of 107 entered the college with higher Cote 
Final scores than the males, but scored lower on the FCI. At the end of the first 
semester there were no significant differences between the Mechanics 101 exam 
scores of males and females. For the cohort of 35 who proceeded directly from 
Mechanics to Electricity and Magnetism there was no significant difference 
between the Cote Final scores of males and females; the Cote Final scores of the 
males who persisted were higher than those in the initial sample as a whole. 
The male students maintained their higher score on the FCI when the measure 
was re-written at the end of the second semester. There was no difference in the 
Electricity and Magnetism exam mark of males and females, but the males who 
persisted had scored higher on the Mechanics final exam than had the females 
who persisted. The results are shown in Tables 5.6 and 5.7. 



Tabic 5.6 

Univariate Tests For Measures 
Cohort Of 107 





M< 


Ue 


Fe 


male 








(n = 


69) 


(n 


= 38) 






Measure 


Mean 


S.D. 


Mean 
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Table 5.7 










Univariate Tests For Measures 
Cohort Of 35 






Measure 


Male 
(n = 22) 
Mean S.D. 


Female 
(n = 13) 
Mean S.D. 


T 


P 


Cote Final 
Force Concept 
Inventory(pre)% 


99.7 
56.1 


9.4 
16.9 


104.0 
40.8 


11.4 
13.4 


1.21 
2.78 


n.s. 
,009 


Phys ICT. exam % 


89.6 


7.9 


79.7 


17.8 


2.24 


.03 


Force Concept 
Invemory(post) 


64.6 


20.4 


43.5 


10.5 


3.45 


.002 


% 

Phys 201 exam % 


71.94 


14.4 


64.2 


28.9 


1.05 


n.s. 



Other recent studies have found that male students achieved higher scores 
on the FCI than female students (Flood, Cross & Snodgrass, 1994; Blue and 
Heller, 1994). Heller has speculated that possible causes for the lower scores of 
females on the inventory is that females frequently do worse than males on 
multiple choice tests because they tend to see nuances of meaning and are not 
prepared to guess in the way males are, and that the classroom climate of high 
school, where boys are asked more questions and the answers of girls are often 
not recognized or valued, inhibits females from ach eving a good understanding 
of the concept of force (Heller, private communication 1994; Jones and Wheatley, 
1990). Another possible cause is that boys frequently have more experience than 
girls in pushing, throwing, and playing with erector sets and mechanical toys. 
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CHAPTER 6 



■ Discussion 

6.0 Introduction 

This exploratory study asked three questions: first, what were the 

■ approaches to learning adopted by physics students; second, what were the 
m intellectual demands of assessment in physics; and third, were there links 

between the approach to learning, the demands of the assessment tasks and 
I the performance of the students. In this chapter the findings are 

summarized, some implications for teaching and learning explored, seme 

■ questions posed, and extensions to this study suggested. 

6.1 Summary Of The Findings 

6.11 Approach To Learning 

_ This study found that incoming physics students approach physics with 

the intention of memorizing formulae rather than understanding concepts, 

■ they adopt Surface or Surface-Achieving Approaches. Those students who 
proceeded directly from the first to the second physics course, i.e. those who 

| were academically on-track, showed a predominantly Achieving or Surface- 

_ Achieving approach on entering the college and by the end of two semesters 

O S9 r 
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of instruction the percentage of those adopting an Achieving Approach had 
declined while the percentage adopting Surface and Deep Approaches had 
increased. The decline in the percentage classified as adopting an Achieving 
Approach, from 31.4% to 2.8%, was as marked as the increase in the Deep 
Approach, from 2.8% to 20%, while the increase in the Surface Approach was 
from 5.7% to 14.3% 

Are the motives and strategies that students adopt the ones that 
teachers consider desirable? If instruction is to be effective it must be aware 
of, and understand, the preconceptions students hold about both the content 
and the learning task in physics. How should teachers use this information 
to counsel students and to guide the form of instruction to better match 
instruction to the beliefs and practices of incoming students, and to the goals 
they, as teachers, consider desirable? 

6.12 Intellectual Demands Of Assessment Items 

The majority of the items of the quizzes, tests, and final exams required 
routine problem solving. For the final examinations of the winter semester 
1994 the percentage of the grade requiring comprehension increased fr» m 
Mechanics 101 (19%) to Electricity and Magnetism 201 (28%) to Waves and 
Optics 301 (32%) and the percentage of items requiring routine problem 
solving declined. No items were coded at the higher levels of Bloom's 
taxonomy (analysis, synthesis, evaluation). Does the physics community 
consider that the levels of thinking demanded by these courses are 
appropriate? Should assessment in physics require students to demonstrate 
analysis, synthesis, and evaluation? 
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A limitation of the present study was that the cognitive level of 
laboratory work was not addressed. In writing a laboratory report are students 
required to complete a table, or to synthesize data, theory and results? Are the 
levels of thinking that were absent from the quizzes, tests and final 
examinations required of students as they carry out and report the results of 
the experiments they undertake? 

6.13 Relationships Between Demands of Assessment, Approach To 
Learning, And Performance 

The findings show that the performance of the students on the final 
examinations was related to the approaches to learning adopted by the 
students. The approach to learning adopted was, in turn, related to the 
cognitive demands of the assessments. In addition there were relationships 
between the prior knowledge of the concept of force and approach to learning 
adopted. 

6.2 Links Between Persistence, Performance And Approach To Learning 

Previous studies have shown that in post secondary science education 
Surface Approach increases and Deep Approach decreases due to the 
pressures of too little time to adequately deal with the amount of content and 
the pace of presentation of material. Concomitantly the research data shows 
that a Deep Approach is related to academic success (Biggs, 1987). 

Teachers must respond to many influences. The students in the study 
entered the college as the new science program was being introduced. For the 
first time most students were entering Mechanics 101 rather than 
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Introductory Physics 111. The instructors were grappling with finding the 
proper level for the course and were very concerned with the demands of the 
course both in terms of the amount of material covered and in terms of the 
level of difficulty of the course. Their solution was to develop quizzes, tests, 
and a final examination that were in their view straightforward. The 
cognitive demands of the final examination were low. Over 90% of the 
questions involved routine problem solving and just 6% required 
comprehension. Seventy seven percent of those who persisted and wrote the 
Mechanics 101 final examination passed. When the successful students 
entered Electricity and Magnetism 201, they were faced with what many 
consider to be the most challenging of the three physics courses (in many 
Cegeps 201 is taught after 301 because of this). The quizzes, tests, and the final 
examination had a much higher fraction of items that required 
comprehension. Correspondingly a higher percentage of those who persisted 
and were successful adopted a Deep Approach. 

The influence of the demands of assessment on the approach to 
learning adopted by the students was further illustrated by the results of the 
step-wise linear regression between grade in the final exam and cognitive 
demands of the exam. There was a negative correlation between Deep 
Approach and the grade in the Mechanics 101 final exam. This can be related 
to over 90% of the items requiring routine problem solving and just 6% 
requiring comprehension. For Electricity and Magnetism 201, there was a 
negative correlation between Surface Approach and grade in the final exam 
and this can be related to the 30% of items on the exam requiring 
comprehension. 

When Pearson product-moment coefficients of correlation were 
calculated between the sub-scale and scale scores of the SPQ and grades in the 
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Mechanics 101 and the Electricity and Magnetism 201 final examinations, a 
negative correlation was found between grade in the Electricity and 
Magnetism 201 final examination and Surface Approach. However there 
were no significant relationships between the Mechanics 101 grade and the 
SPQ scores. 

6.3 Understanding Of The Concept Of Force And Approach To Learning 

The goal of most students is to pass and students adapt and adopt 
practices that they hope will ensure their success, however success in a course 
does not ensure understanding of the material covered. It is accepted that the 
Force Concept Inventory measures understanding of the Newtonian concept 
of force. Over the two semesters the students who persisted showed a gain of 
12% in their score on the FCI. This increase is in agreement with that 
reported in the literature for conventional physics courses, but is much less 
than has been achieved by more interactive courses (Hake, 1994). Such 
courses engage the student in tasks that require active participation and the 
use of higher level thinking skills. The findings of this study that show a 
negative relationship between both Surface and Achieving Approach and 
prior knowledge, as measured by score on the FCI, are in agreement with 
other work in psychology (Beckwith, 1991). 



6.4 Conclusion 



The findings of this work suggest that the intellectual demands of 
assessment tasks influence the approach to learning adopted by students. 
Classroom assessment guides learning. A majority of the questions in the 




quizzes, tests, and final examinations required problem solving with some 
limited understanding of the principles and concepts. It would be possible to 
pass the courses without understanding the concepts. One can infer that the 
students are acquiring content knowledge but must ask if they are able to 
apply this knowledge to complex, unfamiliar situations. 

6*5 Possible Extensions And Questions For Physics Teachers To Consider 

If we are to fully determine the thinking skills developed by current 
instructional practices then as a first step the intellectual demands of 
laboratory work must be determined. Secondly, after designing and testing 
assessment tools that develop higher levels of learning, a study that 
compared the learning outcomes of a control group and a group of students 
that was exposed to assessment that required higher levels of intellectual 
engagement would allow one to explore more fully the links between 
assessment, approach to learning, and performance. Finally, this work has 
shown the relationships that exist between assessment and approach to 
learning for students who persist and succeed, it does not answer directly the 
questions that arise about the relationships between approach to learning and 
assessment for those who drop out and fail. Such a study could help teachers 
give appropriate guidance, and design appropriate instruction to help those 
who currently fail. 

What is known is- that involving students with the tasks rather than 
encouraging silent listening or repetitive calculations does achieve increased 
understanding. However if time is to be devoted to allowing students to 
grapple with the ideas then the content covered must be reduced. (However 
we must ask if the content was covered by the student or by the teacher.) 
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A constant debate among Cegep physics teachers is what topics to 
include and what to omit as they see themselves squeezed between the high 
schools and the universities and buffeted by the changes in curriculum and 
course structures dictated from above. Many traditional practices of teaching 
and assessment are no longer appropriate for the diverse population that fills 
present day physics classrooms. The background, outlook, and needs of 
students have changed. Society no longer accepts without question the value 
of physics as an intellectual discipline and as a subject that can provide 
solutions to societal problems. Faced with these challenges physics teachers 
must re-examine their teaching and assessment methodologies and adopt 
strategies that will encourage meaningful learning of the mix of content and 
process they, the teachers, consider appropriate. What changes, if any, to 
current methods of teaching and assessment will ensure that students 
combine knowledge of current content and concepts with the ability to apply 
these in meaningful ways, and the ability to adapt to as yet unknown 
challenges and ideas? 
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Appendix 1: Taxonomy For Coding The Intellectual Demands Of 
Physics Assessment Items 

MEMORY Recalling information much as it was learned. 

1.1 Memory requires the recall or recognition of specific elements in 
a subject area in a way similar to how it was learned. In its 
simplest form this includes recalling the terminology and 
specific facts associated with an area of subject matter. 

1.2 At a more complex level it means knowing the the major sub- 
areas, methods of inquiry, classifications and ways of thinking 
characteristic of the subject area, as well as the central theories 
and principles. 

Examples: 

(a) Define work. 

(b) State Newton's three laws of motion. 

PROCEDURAL/ ALGORITHMIC Routine calculations 

2.1 Items requiring the student to apply a single principle or to apply a 
predictable formula or algorithm (that requires no choices)to 
generate an answer. The problem is familiar and the rule or 
principle is either given in the problem or very familiar from 
previous rehearsal. 

2.2 Items requiring the student to make a simple interpretation of data. 
Such as read a coordinate from a graph. 

Examples 

(a) If a force of ION acts on a mass of 6kg what is the acceleration? 

(b) What are the X and Y components of a displacement of 6m at an 
angle of 40° to the X axis? 

(c) If Q = Qo(l - e-t/ R C) where R = 100Q,C = 4.7|iF, and 
Qo = 4.5mC, what is the charge on the capacitor after 8.0s? 

(d) Given a position time graph: What is the position at t = 4s? 
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PROCEDURAL/COMPREHENSION Problem solving with limited 

comprehension 



3.1 Solving a problem that requires that choices be made about 
which rule or principle to apply based on the information given 
in the problem. The situation is familiar. The problem 
involves a familiar pattern, and problems of similar type have 
been rehearsed either in class, during quizes or tests, or in 
assignments. 

3.2 Items requiring the student to make straightforward 
interpretations of data requiring some limited judgement. Such 
as construct the tangent at a point on a curved position-time . 
graph and take the slope to find the instantanous velocity. 
Produce a routine free body diagram. 

Examples 

(a) A dog is running at a constant velocity of 4.0m/s towards a cat. 
When the dog is 16.0m behind the cat the cat starts accelerating 
from rest at 0.30m/s 2 . When and where does the dog catch up 
with the cat? 

(b) A block of mass 5.0kg rests on a smooth horizontal surface. A 
force of 4.0N acts parallel to the surface. Construct the free body 
diagram of the block. 

(c) Given a curved position time graph. Find the instantaneous 
velocity at some given time. 

COMPREHENSION Reporting information in a way other than which it 

was learned in order to demonstrate that it has been 
understood. Apply a rule or principle in such a way 
as to demonstrate not only the application of the 
rule but an understanding of why the application is 
appropriate. 

4.1 State a principle and give an example of its application. Describe 
properties, units, vector/number, typical magnitudes. 

4.2 Interpret a principle in different ways: symbols, words, graphs, 
numbers. 

4.3 In a physical situation identify those relations which are 
applicable and use them without confusion. 
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Comprehension continued 
Examples: 

(a) Given a data table of y(t) for a ball thrown into the air construct 
the velocity time graph. 

(b) Given a velocity time graph having a changing velocity, 
construct the position time graph. 

(c) A boat is sailing on Lac St Louis with a velocity of vi = (5m/s, 
323°), ten minutes later it has a velocity of V2 = (- 6.0, 2.0)m/s. 
Sketch, not to scale, the vector triangle showing the vectors vi, 

V2, and Av. 

(d) A singly charged positive ion, mass, 2.5 x 10^k^, enters at right 
angles to a uniform magnetic field with magnitude B = 5.0 x 10" 
^T. What must be the magnitude of the perpendicular electric 
field, E, which would allow the ion to pass through without 
being deflected? Include a diagram showing the relative 
orientations of V, B, E, Fmag/ Fe- 
te) A car is driving around a banked curve ( r = 1000m, angle 7°) 

The coefficient of kinetic friction between the road and the tires 
is 0.4 and the car is driven at the maximum posible speed 
without slipping sideways. Draw the free body diagram of the 
car. 



ANALYSIS Taking learned information apart. 

5.1 Analysis refers to logic, induction, deduction, and formal 
reasoning: it is the breakdown of a problem into its constituent 
ideas or parts so that the relative hierachy of the ideas is made 
clear and/or the relations between the ideas is made explicit. 

5.2 Learning outcomes represent a higher intellectual level than 
comprehension because they require an understanding of not 
only the content and the structural relationships of the material 
but the ability to apply this knowledge to a new, unfamiliar, 
situation. The learner must be able to identify the important 
elements in a problem, determine the connections between the 
parts, then recognize the stucture and principles that connect the 
situation. 

5.3 The student is asked to distinguish, classify, and relate the 
assumptions, hypotheses, data, conclusions and structure of the 
question. 
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6 SYNTHESIS 



Create a product that is new. Draw together 
elements and parts to form a whole that is new. 



6.1 Synthesis is the putting together of elements and parts to form a 
whole. This involves arranging and combining pieces in such a 
way as to create a structure that was not there before. 

6.2 Students create, integrate, and combine ideas and data into a 
product that is new to them. 

7 EVALUATION Judging the value of something for a particular 

purpose. Use of a standard of appraisal. The 
criteria may be determined or given. Evaluation 
has two steps. The first is to set up appropriate 
standards(criteria) and the second is to determine 
how closely the object or idea meets these 
standards. 

7.1 Evaluation of the accuracy of ideas, works, solutions, methods or 
material, from logical accuracy, consistency or other internal 
criteria. 

7.2 Evaluation of material with respect to remembered or specified 
criteria. Evaluation requires that the student makes judgements 
about something he or she knows, analyses, synthesizes, and so 
forth, on the basis of criteria which can be made explicit. 



ERIC 
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Appendix 2: 



Univariate Analysis Of SPQ Sub-Scale And Scale Scores 
(n = 35) 





Pre 




Post 










Mean 


b.D. 


Mean 


S.D. 


T 


P 


bl \2 sub-scales 














Surface motive 


24.54 


4.22 


26.26 


4.46 


2.09 


.04 


Surface strategy 


22.43 


3.89 


23.06 


4.07 




ii.j. 


Deep motive 


22.74 


4.24 


21.17 


5.23 


1.72 


.09 


Deep strategy 


22.17 


3.87 


22.17 


3.85 


0 


n.s. 


Achieving motive 


26.17 


3.83 


24.82 


5.38 


1.96 


.05 


Achieving strategy 


22.57 


5.04 


21.11 


5.79 


1.83 


.07 


SPQ Scales 














Surface approach 


46.97 


7.59 


49.31 


7.43 


1.78 


.08 


Deep approach 


44.91 


6.83 


43.34 


7.51 


1.16 


n.s. 


Achieving approach 


48.74 


7.51 


45.94 


9.64 


2.26 


.03 



Note The SPQ was written at the start of the first semester (pre) and the end 

of the second semester (post)? 



Appendix 3: Percent Of Items By Course And Level Of Thinking 







Level 


of Thinking 




Course 


Memory 


Procedural/ 
Algorithmic 


Procedural/ 
Comprehension 


Comprehension 


101 


0.8 


11.5 


64.8 


22.9 


201 




13 


57 


30 


301 


2 


13.5 


51.5 


33 



Note The items are expressed as as a percentage of the number of items in each 

course; 

Course 10! 253 items coded 

201 236 items coded 
301 221 items coded 
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Appendix 4: Percentage Of Items And Percentage Of Grade By Level Of 
Thinking: John Abbott College Winter Semester 1994 









Level of Thinking 








Memory 


Procedural/ 


Procedural/ 


Comprehension 








Algorithmic 


Comprehension 




Course 










101 


0.67 


16.1 


62.4 


20.8 


Items 


201 




11.2 


54.3 


34.4 


% 


301 


1 


22.3 


42.7 


33.9 




101 


0.32 


8.5 


51.6 


19.2 


Grade 


201 




5.2 


46.5 


28.4 


% 


301 


0.2 


5.8 


42.2 


32 



NotGS (a) The marks are expressed as a percentages of the grade of 100 that was assigned 
to the course. The quizzes, tests, and final contributed 80% of the grade, 
(b) The items are expressed as as a percentage of the number of items in each 
course; 

Course 101 148 items coded 
201 116 items coded 
301 103 items coded 

Appendix 5: Contributions Of Quizzes, Tests, And Final Examination 

To Grade By Level Of Thinking 











Level of thinking 




Course 






Memory 


Procedural 


Procedural 


Comprehension 










Algorithmic 


Comprehensio 






Quiz 


(10) 




1.6 


n 

6.8 


1.6 


101 


Tests 


(30) 


0.3 


4.2 


18.6 


6.7 




Final 


(40) 




2.6 


26.2 


10.9 




Sum 


(80) 


0.3 


8.5 


51.6 


19.2 




Quiz 


(10) 




0.5 


6.0 


3.4 


201 


Tests 


(30) 




2.0 


17.2 


10.9 




Final 


(40) 




2.6 


23 2 


13.9 




Sum 


(80) 




5.2 


46.5 


28.3 




Quiz 


(10) 


0.2 


2.1 


5.5 


2.4 


301 


Tests 


(30) 




2.3 


13.9 


13.8 




Final 


(40) 




1.5 


22.8 


15.8 




Sum 


(80) 


0.2 


5.9 


42.2 


32.0 
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Appendix 6: Pearson Product-Moment Coefficients Of Correlation 

Between SPQ Sub-Scales And Scales And Physics 101 And 
201 Final Examination Grades 





rhysies HJl 


rnysics iui 


1 IiySKS ZU1 




(n = 107) 


(n = 35) 


(n = 35) 




grade p 


grade p 


grade p 


SPQ sub-scales 








Surface motive 


0.098 


-0.056 


-0.289 


Surface strategy 


0.067 


-0.091 


-0.292 


Deep motive 


0.010 


0.054 


0.056 


Deep Strategy 


-0.044 


0.061 


-0.131 


Achieving motive 


0.124 


0.049 


0.181 


Achieving strategy 


0.070 


0.145 


0.035 


SPQ Scales 








Surface approach 


0.094 


-0.078 


-0.333 .51 


Deep approach 


-0.019 


0.068 


0.028 


Achieving approach 


0.113 


0.122 


0.087 



Appendix 7: Pearson Product-Moment Coefficients Of Correlation 

Between SPQ Sub-Scale And Scale Scores And Score On 
The FCI 





Pretest 


Pretest 


Posttest 




(n 


= 107) 


(n 


= 35) 


(n 


= 35) 




score 


P 


score 


P 


score 


P 


SPQ sub-scales 














Surface motive 


-0.107 




-0.095 




- 0.296 


.08 


Surface strategy 


-0.124 




-0.314 


.06 


- 0.477 


.004 


Deep motive 


0.034 




-0.049 




-0.052 




Deep Strategy 


0.114 




0.204 




0.061 




Achieving motive 


-0.217 


.02 


-0.194 




-0.184 




Achieving strategy 


-0.313 


< .001 


-0.349 


.04 


- 0.468 


.005 


SPQ Scales 














Surface approach 


-0.128 




-0.214 




- 0.438 


.008 


Deep approach 


0.084 




0.085 




-0.005 


.02 


Achieving approach 


-0.318 


< .001 


-0.333 


.05 


- 0.384 



Notes: Probabilities not shown are not significant. 

Physics 101 results are for the SPQ written at the start of the first semester 
Physics 201 results are for the SPQ written at the end of the second semester 
Pretest results are for the SPQ and FCI written at the start of the first semester 
Posttest results are for the SPQ and FCI written at the end of the second semester 
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Appendix 8: Univariate Analysis Of FCI And SPQ Sub-Scale And Scale 

Scores (n = 35) 





Pre 




Post 










Mean 


S.D. 


Mean 


S.D. 


T 


P 


SPQ sub-scales 














Surface? motivo 


24.54 


4.22 


26.26 


4.46 


2.09 


.04 


Surface strategy 


22.43 


3.89 


23.06 


4.07 


0.98 


n.s. 


Deep motive 


22.74 


4.24 


21.17 


5.23 


1.72 


.09 


Deep strategy 


22.17 


3.87 


22.17 


3.85 


0 


n.s. 


Achieving motive 


26.17 


3.83 


24.82 


5.38 


1.96 


.05 


Achieving strategy 


22.57 


5.04 


21.11 


5.79 


1.83 


.07 


SPQ Scales 














Surface approach 


46.97 


7.59 


49.31 


7.43 


1.78 


.08 


Deep approach 


44.91 


6.83 


43.34 


7.51 


1.16 


n.s. 


Achieving approach 


48.74 


7.51 


45.94 


9.64 


2.26 


.03 


FCI % 


50.4 


17.2 


56.7 


20.0 


3.416 


.002 



Appendix 9: Pearson Product-Moment Coefficients Of Correlation 

Between High School Performance (Cote Final), Prior 
Knowledge (FCI), And Physics 101 And 201 Final Exam 
Grades 





Physics 101 


Physics 101 


Physics 201 




(n = 107) 


(n = 35) 


(n = 35) 




grade p 


grade p 


grade p 


CF 


0.511 .000 


0.306 .07 


0.461 .000 


FCI1 


0.311 .1X11 


0.437 .009 


0.370 .03 


FCI2 






0.391 .02 



Note FCI1 measure written at the start of the first semester. 

FCI2 measure written at the end of the second semester. 
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